Spatially adaptive quantization-aware deblocking filter

ABSTRACT

A spatially adaptive quantization-aware deblocking filter is used for encoding or decoding video or image frames. The deblocking filter receives a reconstructed frame produced based on dequantized and inverse transformed coefficients of a video frame or an image frame. The reconstructed frame is filtered according to adaptive quantization field data for the video or image frame. The adaptive quantization field data represents weights applied to quantization values used at different areas of the video or image frame. A number of blocking artifacts remaining within the resulting filtered frame is determined. The adaptive quantization field data is then adjusted based on that number of blocking artifacts. The filtered frame is then filtered according to the adjusted adaptive quantization field data. The resulting re-filtered frame is then output to an output source, such as for transmission, display, storage, or further processing.

BACKGROUND

Digital video streams may represent video using a sequence of frames orstill images. Digital video can be used for various applicationsincluding, for example, video conferencing, high definition videoentertainment, video advertisements, or sharing of user-generatedvideos. A digital video stream can contain a large amount of data andconsume a significant amount of computing or communication resources ofa computing device for processing, transmission, or storage of the videodata. Various approaches have been proposed to reduce the amount of datain video streams, including encoding or decoding techniques.

The approaches for reducing the amount of data in video streams may alsobe used to reduce the amount of data in an image. Image contentrepresents a significant amount of online content. A web page mayinclude multiple images, and a large portion of the time and resourcesspent rendering the web page are dedicated to rendering those images fordisplay. The amount of time and resources required to receive and renderan image for display depends in part on the manner in which the image iscompressed. As such, an image, and therefore a web page that includesthe image, can be rendered faster by reducing the total data size of theimage using encoding and decoding techniques.

SUMMARY

Disclosed herein are, inter alia, systems and techniques for video orimage coding using a spatially adaptive quantization-aware deblockingfilter.

A method for decoding an encoded frame from a bitstream according to animplementation of this disclosure comprises decoding syntax dataassociated with the encoded frame from the bitstream. The syntax datainclude quantized transform coefficients of encoded blocks of theencoded frame and adaptive quantization field data representing weightsapplied to quantization values used to encode the encoded blocks. Themethod further comprises dequantizing and inverse transforming thequantized transform coefficients of the encoded blocks to producedecoded blocks. The method further comprises reconstructing the decodedblocks into a reconstructed frame. The method further comprises applyinga deblocking filter to the reconstructed frame according to the adaptivequantization field data to produce a first filtered frame. The methodfurther comprises determining a number of blocking artifacts within thefirst filtered frame. The method further comprises adjusting the atleast some of the adaptive quantization field data based on the numberof blocking artifacts to produce adjusted adaptive quantization fielddata. The method further comprises applying the deblocking filter to thefirst filtered frame according to the adjusted adaptive quantizationfield data to produce a second filtered frame. The method furthercomprises outputting the second filtered frame for display.

An apparatus for decoding an encoded frame from a bitstream according toan implementation of this disclosure comprises a processor configured toexecute instructions stored in a non-transitory storage medium. Theinstructions include instructions to decode encoded blocks of theencoded frame to produce decoded blocks. The instructions furtherinclude instructions to reconstruct the decoded blocks into areconstructed frame. The instructions further include instructions tofilter the reconstructed frame according to adaptive quantization fielddata associated with the encoded frame to produce a first filteredframe. The instructions further include instructions to subsequent tofiltering the reconstructed frame, use a psychovisual model to adjust atleast some of the adaptive quantization field data. The instructionsfurther include instructions to subsequent to adjusting the at leastsome of the adaptive quantization field data, filter the first filteredframe according to the adaptive quantization field data to produce asecond filtered frame. The instructions further include instructions tooutput the second filtered frame for display.

A method according to an implementation of this disclosure comprisesapplying first filter parameters to a reconstructed frame to produce afirst filtered frame. The first filter parameters are defined based onadaptive quantization field data associated with the reconstructedframe. The method further comprises adjusting the first filterparameters to produce second filter parameters. The adjusting includesincreasing at least one value of the adaptive quantization field data.The method further comprises applying the second filter parameters tothe first filtered frame to produce a second filtered frame. The methodfurther comprises outputting the second filtered frame for display.

These and other aspects of this disclosure are disclosed in thefollowing detailed description of the implementations, the appendedclaims, and the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The description herein makes reference to the accompanying drawingsdescribed below, wherein like reference numerals refer to like partsthroughout the several views.

FIG. 1 is a schematic of an example of an encoding and decoding system.

FIG. 2 is a block diagram of an example of a computing device that canimplement a transmitting station or a receiving station of an encodingand decoding system.

FIG. 3A is a diagram of an example of a video stream to be encoded andsubsequently decoded.

FIG. 3B is a diagram of an example of an image to be encoded andsubsequently decoded.

FIG. 4 is a block diagram of an example of an encoder.

FIG. 5 is a block diagram of an example of a decoder.

FIG. 6 is a block diagram of an example of a spatially adaptivequantization-aware deblocking filter used for encoding or decoding avideo frame or an image.

FIG. 7 is a block diagram representing portions of a video frame orimage.

FIG. 8 is a flowchart diagram of an example of a technique for encodingor decoding a video frame or an image frame using a spatially adaptivequantization-aware deblocking filter.

FIG. 9 is a flowchart diagram of an example of a technique foriteratively filtering a video frame or an image frame according toadaptive quantization field data.

FIG. 10 is an illustration of examples of reproductions of an originalvideo frame or image using different filtering or filter-lesstechniques.

DETAILED DESCRIPTION

Lossy encoding involves reducing the amount of data within an image or avideo to be encoded, such as using quantization. In exchange for adecreased bit cost of the resulting encoded image or video, the imagesuffers certain quality loss. The extent of the quality loss dependslargely upon the manner by which the image data or the video data wasquantized during the encoding. In particular, the quantization of imagedata or video data can result in discontinuities along block boundaries,such as blocking artifacts. The quantization error resulting from lossyencoding is typically indicative of the amount of blocking artifactsresulting from the encoding. As such, the greater the quantizationerror, the greater the number of blocking artifacts, and, therefore, thegreater the quality loss.

Blocking artifacts may be reduced by applying a filter, such as adeblocking filter, to the coefficients of a video block or of an imageblock. The deblocking filter may be applied to a reconstructed frame ora portion of a reconstructed frame at the end of a reconstructing phasein the encoding process or at the end of the decoding process. Thedeblocking filter removes blocking artifacts from a frame to reproducethat frame in its pre-encoded form. However, a typical deblocking filterdoes not have visibility into the amount of quantization that was usedto encode a given frame or the specific blocks therein. As such, andparticularly where different quantization levels are used for differentblocks within a single frame, the deblocking filter may use a suboptimalor inappropriate filtering strength or filter radius size.

Implementations of this disclosure address problems such as these usinga deblocking filter controlled based on adaptive quantization fielddata. The adaptive quantization field data is used to control thestrength and spatial size of the deblocking filter. As such, thedeblocking filter uses a greater strength and/or spatial size for blocksthat the adaptive quantization field data indicates were encoded at agreater quantization level and a lesser strength and/or spatial size forblocks that the adaptive quantization field data indicates were encodedat a lesser quantization level. Fine details within a frame arepreserved by controlling the strength and spatial size of the filteringbased on the adaptive quantization field data. A psychovisual model isused in connection with the filtering to determine whether adjustmentsshould be made to the adaptive quantization field data. For example, ifa number of blocking artifacts remaining within an area of the frame toencode or decode is too large, the psychovisual model can indicate toincrease vales of the adaptive quantization field data for that area.

Further details of techniques for video or image coding using aspatially adaptive quantization-aware deblocking filter are describedherein with initial reference to a system in which they can beimplemented. FIG. 1 is a schematic of an example of an encoding anddecoding system 100. The encoding and decoding system 100 includes atransmitting station 102, a receiving station 104, and a network 106.

The transmitting station 102 is a computing device that encodes andtransmits a video or an image. Alternatively, the transmitting station102 may include two or more distributed computing devices for encodingand transmitting a video or an image. The receiving station 104 is acomputing device that receives and decodes an encoded video or anencoded image. Alternatively, the receiving station 104 may include twoor more distributed computing devices for receiving and decoding anencoded video or an encoded image. An example of a computing device usedto implement one or both of the transmitting station 102 or thereceiving station 104 is described below with respect to FIG. 2.

The network 106 connects the transmitting station 102 and the receivingstation 104 for the encoding, transmission, receipt, and decoding of avideo or an image. The network 106 can be, for example, the Internet.The network 106 can also be a local area network, a wide area network, avirtual private network, a cellular telephone network, or another meansof transferring the video or the image from the transmitting station 102to the receiving station 104.

Implementations of the encoding and decoding system 100 may differ fromwhat is shown and described with respect to FIG. 1. In someimplementations, the encoding and decoding system 100 can omit thenetwork 106. In some implementations, a video or an image can be encodedand then stored for transmission at a later time to the receivingstation 104 or another device having memory. In some implementations,the receiving station 104 can receive (e.g., via the network 106, acomputer bus, and/or some communication pathway) the encoded video orencoded image and store the encoded video or encoded image for laterdecoding.

In some implementations, the functionality of the transmitting station102 and of the receiving station 104 can change based on the particularoperations performed. For example, during operations for encoding avideo or an image, the transmitting station 102 can be a computingdevice used to upload the video or the image to be encoded to a server,and the receiving station 104 can be the server that receives the videoor the image from the transmitting station 102 and encodes the video orthe image for later use (e.g., in storing a bitstream, rendering awebpage, or the like).

In another example, during operations for decoding an encoded video orencoded image, the transmitting station 102 can be a server that decodesthe encoded video or encoded image, and the receiving station 104 can bea computing device that receives the decoded video or decoded image fromthe transmitting station 102 and outputs or renders the decoded video ordecoded image (e.g., to an output video stream, as part of a webpage, orthe like).

In some implementations, the encoding and decoding system 100 may omitthe network 106. In some implementations, a transport protocol may beused to transport a video or an image, or an encoded video or encodedimage, over the network 106. For example, the transport protocol may bethe real-time transport protocol, the hypertext transfer protocol, oranother image or video streaming protocol.

In some implementations, each of the transmitting station 102 and thereceiving station 104 may include functionality for both encoding anddecoding a video or an image. For example, the encoding and decodingsystem 100 may be implemented using a video conferencing system. Thereceiving station 104 may be a computing device of a video conferenceparticipant. The receiving station 104 may receive an encoded videobitstream from a video conference server (e.g., the transmitting station102) to decode and view. The receiving station 104 may further encodeand transmit another video bitstream to the video conference server,such as for decoding and viewing by computing devices of other videoconference participants.

FIG. 2 is a block diagram of an example of a computing device 200 thatcan implement a transmitting station or a receiving station of anencoding and decoding system, such as the encoding and decoding system100 shown in FIG. 1. For example, the computing device 200 can implementone or both of the transmitting station 102 or the receiving station 104shown in FIG. 1. The computing device 200 can be in the form of acomputing system including multiple computing devices or in the form ofone computing device. For example, the computing device 200 can be oneof a mobile phone, a tablet computer, a laptop computer, a notebookcomputer, a desktop computer, a server computer, a game console, awearable device, or the like.

A processor 202 in the computing device 200 can be a conventionalcentral processing unit. Alternatively, the processor 202 can be anothertype of device, or multiple devices, now existing or hereafterdeveloped, capable of manipulating or processing information. Forexample, although the disclosed implementations can be practiced withone processor as shown (e.g., the processor 202), advantages in speedand efficiency can be achieved by using more than one processor.

A memory 204 in the computing device 200 can be a read-only memorydevice or a random-access memory device in an implementation. However,other suitable types of storage devices can be used as the memory 204.The memory 204 can include code and data 206 that is accessed by theprocessor 202 using a bus 212. The memory 204 can further include anoperating system 208 and application programs 210, the applicationprograms 210 including at least one program that permits the processor202 to perform the techniques described herein. For example, theapplication programs 210 can include applications 1 through N, whichfurther include video or image coding software that performs some or allof the techniques described herein. The computing device 200 can alsoinclude a secondary storage 214, which can, for example, be a memorycard used with a mobile computing device. For example, an image can bestored in whole or in part in the secondary storage 214 and loaded intothe memory 204 as needed for processing.

The computing device 200 can also include one or more output devices,such as a display 218. The display 218 may be, in one example, atouch-sensitive display that combines a display with a touch-sensitiveelement that is operable to sense touch inputs. The display 218 can becoupled to the processor 202 via the bus 212. Other output devices thatpermit a user to program or otherwise use the computing device 200 canbe provided in addition to or as an alternative to the display 218. Whenthe output device is or includes a display, the display can beimplemented in various ways, including as a liquid crystal display, acathode-ray tube display, or a light emitting diode display, such as anorganic light emitting diode display.

The computing device 200 can also include or be in communication with animage-sensing device 220, for example, a camera, or anotherimage-sensing device, now existing or hereafter developed, which cansense an image such as the image of a user operating the computingdevice 200. The image-sensing device 220 can be positioned such that itis directed toward the user operating the computing device 200. Forexample, the position and optical axis of the image-sensing device 220can be configured such that the field of vision includes an area that isdirectly adjacent to the display 218 and from which the display 218 isvisible.

The computing device 200 can also include or be in communication with asound-sensing device 222, for example, a microphone or anothersound-sensing device, now existing or hereafter developed, which cansense sounds near the computing device 200. The sound-sensing device 222can be positioned such that it is directed toward the user operating thecomputing device 200 and can be configured to receive sounds, forexample, speech or other utterances, made by the user while the useroperates the computing device 200.

Implementations of the computing device 200 may differ from what isshown and described with respect to FIG. 2. In some implementations, theoperations of the processor 202 can be distributed across multiplemachines (wherein individual machines can have one or more processors)that can be coupled directly or across a local area or other network. Insome implementations, the memory 204 can be distributed across multiplemachines, such as a network-based memory or memory in multiple machinesperforming the operations of the computing device 200. In someimplementations, the bus 212 of the computing device 200 can be composedof multiple buses. In some implementations, the secondary storage 214can be directly coupled to the other components of the computing device200 or can be accessed via a network and can comprise an integratedunit, such as a memory card, or multiple units, such as multiple memorycards.

FIG. 3A is a diagram of an example of a video stream 300 to be encodedand subsequently decoded. The video stream 300 represents a typicalvideo stream that can be encoded into a compressed bitstream, forexample, using the transmitting station 102 shown in FIG. 1, andsubsequently decoded into an output video stream, for example, using thereceiving station 104 shown in FIG. 1.

The video stream 300 includes a video sequence 302. At the next level,the video sequence 302 includes a number of adjacent frames 304. Whilethree frames are depicted as the adjacent frames 304, the video sequence302 can include another number of adjacent frames 304. The adjacentframes 304 can then be further subdivided into individual frames, forexample, a frame 306. At the next level, the frame 306 can be dividedinto a series of planes or segments 308. The segments 308 can be subsetsof frames that permit parallel processing, for example. The segments 308can also be subsets of frames that can separate the video data intoseparate colors. For example, a frame 306 of color video data caninclude a luminance plane and two chrominance planes. The segments 308may be sampled at different resolutions.

Whether or not the frame 306 is divided into segments 308, the frame 306may be further subdivided into blocks 310, which can contain datacorresponding to, for example, 16×16 pixels in the frame 306. The blocks310 can also be arranged to include data from one or more segments 308of pixel data. The blocks 310 can also be of any other suitable sizesuch as 4×4 pixels, 8×8 pixels, 16×8 pixels, 8×16 pixels, 16×16 pixels,or larger. Unless otherwise noted, the terms block and macroblock areused interchangeably herein.

FIG. 3B is a diagram of an example of an image frame 312 to be encodedand subsequently decoded. The image frame 312 represents a typical imagethat can be encoded into a compressed bitstream or storage, for example,using the transmitting station 102 shown in FIG. 1, and subsequentlydecoded for rendering at a display, for example, using the receivingstation 104 shown in FIG. 1. The image frame 312 may have the same formas the frame 306 shown in FIG. 3A. For example, the image frame 312 maybe divided into the image segments 314 and/or the image blocks 316 forfurther processing during encoding or decoding.

FIG. 4 is a block diagram of an encoder 400. The encoder 400 can beimplemented, as described above, in the transmitting station 102 shownin FIG. 1, such as by providing a computer software program stored inmemory, for example, the memory 204 shown in FIG. 2. The computersoftware program can include machine instructions that, when executed bya processor such as the processor 202 shown in FIG. 2, cause thetransmitting station 102 to encode video data in the manner described inFIG. 4. The encoder 400 can also be implemented as specialized hardware(e.g., an integrated circuit) included in, for example, the transmittingstation 102. In some implementations, the encoder 400 is a hardwareencoder.

The encoder 400 has the following stages to perform the variousfunctions in a forward path (shown by the solid connection lines) toproduce an encoded or compressed bitstream 420 using the video stream300 as input: an intra/inter prediction stage 402, a transform stage404, a quantization stage 406, and an entropy encoding stage 408. Theencoder 400 may also include a reconstruction path (shown by the dottedconnection lines) to reconstruct a frame for encoding of future blocks.In FIG. 4, the encoder 400 has the following stages to perform thevarious functions in the reconstruction path: a dequantization stage410, an inverse transform stage 412, a reconstruction stage 414, and adeblocking filter stage 416. Other structural variations of the encoder400 can be used to encode the video stream 300.

When the video stream 300 is presented for encoding, respective adjacentframes 304, such as the frame 306, can be processed in units of blocks.At the intra/inter prediction stage 402, respective blocks can beencoded using intra-frame prediction (also called intra-prediction) orinter-frame prediction (also called inter-prediction). In any case, aprediction block can be formed. In the case of intra-prediction, aprediction block may be formed from samples in the current frame thathave been previously encoded and reconstructed. In the case ofinter-prediction, a prediction block may be formed from samples in oneor more previously constructed reference frames.

Next, the prediction block can be subtracted from the current block atthe intra/inter prediction stage 402 to produce a residual block (alsocalled a residual or prediction residual). The transform stage 404transforms the residual into transform coefficients in, for example, thefrequency domain using block-based transforms. The quantization stage406 converts the transform coefficients into discrete quantum values,which are referred to as quantized transform coefficients, using aquantizer value or a quantization level. For example, the transformcoefficients may be divided by the quantizer value and truncated.

The quantized transform coefficients are then entropy encoded by theentropy encoding stage 408. The entropy-encoded coefficients, togetherwith other information used to decode the block (which may include, forexample, syntax elements such as used to indicate the type of predictionused, transform type, motion vectors, a quantizer value, or the like),are then output to the compressed bitstream 420. The compressedbitstream 420 can be formatted using various techniques, such asvariable length coding (VLC) or arithmetic coding. The compressedbitstream 420 can also be referred to as an encoded video stream orencoded video bitstream, and the terms will be used interchangeablyherein.

The reconstruction path (shown by the dotted connection lines) can beused to ensure that the encoder 400 and a decoder 500 (described belowwith respect to FIG. 5) use the same reference frames to decode thecompressed bitstream 420. The reconstruction path performs functionsthat are similar to functions that take place during the decodingprocess (described below with respect to FIG. 5), including dequantizingthe quantized transform coefficients at the dequantization stage 410 andinverse transforming the dequantized transform coefficients at theinverse transform stage 412 to produce a derivative residual block (alsocalled a derivative residual).

At the reconstruction stage 414, the prediction block that was predictedat the intra/inter prediction stage 402 can be added to the derivativeresidual to create a reconstructed block. The deblocking filter stage416 can be applied to the reconstructed block to reduce distortion suchas blocking artifacts. Implementations and examples of a deblockingfilter used at the deblocking filter stage 416 are described below withrespect to FIG. 6.

Implementations of the encoder 400 may differ from what is shown anddescribed with respect to FIG. 4. In particular, the encoder 400 asshown in FIG. 4 is an example of an encoder for encoding video data,such as a video frame or a video block. However, in otherimplementations, the encoder 400 may be an example of an encoder forencoding image data, such as an image frame or an image block.

In such an implementation, the encoder 400 omits the intra/interprediction stage 402. For example, the input image data is firstprocessed at the transform stage 404 and then the quantization stage 406before it enters the reconstruction path of stages 410, 412, 414, and416. The output of the deblocking filter stage 416 can be sent to thetransform stage 404 for further processing. If the reconstruction pathis not needed, such as because the error level for the image frame meetsa threshold, the reconstruction path may not be followed and thequantized image data may instead proceed to the entropy encoding stage408 and then output to the compressed bitstream 420. In someimplementations where the encoder 400 is used to encode image data, theencoder 400 may omit the entropy encoding stage 408.

In some implementations, the encoder 400 may be a non-transform basedencoder for image or video coding. In such an implementation, theencoder 400 can quantize the residual signal directly without thetransform stage 404 for certain blocks or frames. In someimplementations, the quantization stage 406 and the dequantization stage410 may be combined into a common stage.

FIG. 5 is a block diagram of a decoder 500. The decoder 500 can beimplemented, as described above, in the receiving station 104 shown inFIG. 1, such as by providing a computer software program stored inmemory, for example, the memory 204 shown in FIG. 2. The computersoftware program can include machine instructions that, when executed bya processor such as the processor 202 shown in FIG. 2, cause thereceiving station 104 to decode video data in the manner described inFIG. 5. The decoder 500 can also be implemented in specialized hardware(e.g., an integrated circuit) included in, for example, the receivingstation 104.

The decoder 500, similar to the reconstruction path of the encoder 400described above, includes in one example the following stages to performvarious functions to produce an output video stream 516 from thecompressed bitstream 420: an entropy decoding stage 502, adequantization stage 504, an inverse transform stage 506, an intra/interprediction stage 508, a reconstruction stage 510, a deblocking filterstage 512, and an optional post-filtering stage 514. Other structuralvariations of the decoder 500 can be used to decode the compressedbitstream 420.

When the compressed bitstream 420 is presented for decoding, the dataelements within the compressed bitstream 420 can be decoded by theentropy decoding stage 502 to produce a set of quantized transformcoefficients. The dequantization stage 504 dequantizes the quantizedtransform coefficients (e.g., by multiplying the quantized transformcoefficients by the quantizer value), and the inverse transform stage506 inverse transforms the dequantized transform coefficients to producea derivative residual that can be identical to that created by theinverse transform stage 412 in the encoder 400. Using header informationdecoded from the compressed bitstream 420, the decoder 500 can use theintra/inter prediction stage 508 to create the same prediction block aswas created in the encoder 400 (e.g., at the intra/inter predictionstage 402).

At the reconstruction stage 510, the prediction block can be added tothe derivative residual to create a reconstructed block. The deblockingfilter stage 512 can be applied to the reconstructed block to reduceblocking artifacts (e.g., using deblocking filtering, sample adaptiveoffset filtering, other loop filter functionality, or the like, or acombination thereof). Implementations and examples of a deblockingfilter used at the deblocking filter stage 512 are described below withrespect to FIG. 6. Other filtering can be applied to the reconstructedblock. In this example, the post-filtering stage 514 is applied to thereconstructed block to reduce blocking distortion, and the result isoutput as the output video stream 516. The output video stream 516 canalso be referred to as a decoded video stream, and the terms will beused interchangeably herein.

Implementations of the decoder 500 may differ from what is shown anddescribed with respect to FIG. 5. In some implementations, the decoder500 can omit or otherwise not process data using the post-filteringstage 514. Further, while the decoder 500 as shown in FIG. 5 is anexample of a decoder for decoding encoded video data, such as an encodedvideo frame or an encoded video block, in other implementations, thedecoder 500 may be an example of a decoder for decoding encoded imagedata, such as an encoded image frame or an encoded image block.

In such an implementation, the decoder 500 omits the intra/interprediction stage 508. For example, the output of the deblocking filterstage 512 can be sent to the dequantization stage 504 for furtherprocessing. In another example, where the error level for the imageframe meets a threshold, the output of the deblocking filter stage 512can instead be sent to the post-filtering stage 514 and/or as output toa display for rendering. In some implementations where the decoder 500is used to decode encoded image data, the decoder 500 may omit theentropy decoding stage 502.

FIG. 6 is a block diagram of an example of a spatially adaptivequantization-aware deblocking filter 600 (hereafter referred to as thedeblocking filter 600) used for encoding or decoding a frame, such as avideo frame or an image frame. The deblocking filter 600 may, forexample, be the deblocking filter used in the deblocking filter stage416 shown in FIG. 4, the deblocking filter using in the deblockingfilter stage 512 shown in FIG. 5, or both.

The deblocking filter 600 receives reconstructed frame and adaptivequantization field data 602 and uses those to produce a filtered frame604. The reconstructed frame includes data produced by reconstructingsome number of decoded blocks, which decoded blocks representdequantized and inverse transformed coefficients. For example, thereconstructed frame can be produced using the reconstruction stage 414shown in FIG. 4 or the reconstruction stage 510 shown in FIG. 5. Theadaptive quantization field data represents weights applied toquantization values used to encode those blocks. The adaptivequantization field data includes values indicative of the quantizationlevels used to encode different areas of the frame. The values of theadaptive quantization field data are determined by analyzing the entireframe, such as to determine the areas in which to apply a higher orlower weight to the quantization.

The deblocking filter 600 includes an artifact removal stage 606, apsychovisual modeling stage 608, and a parameter adjustment stage 610.The artifact removal stage 606 receives the reconstructed frame andadaptive quantization field data 602. The artifact removal stage 606filters the reconstructed frame to remove some number of blockingartifacts from the reconstructed frame. The filtering of thereconstructed frame by the artifact removal stage 606 is controlledusing the adaptive quantization field data.

The artifact removal stage 606 compares the reconstructed frame to theoriginal (e.g., pre-encoded) frame to determine differences betweenthose frames as a result of the encoding. In areas where thosedifferences reflect that the reconstructed frame does not accuratelyrepresent the original frame, the artifact removal stage 606 applies afilter to reduce the number of blocking artifacts. The artifact removalstage 606 modules a transition from a small difference to a largedifference based on the adaptive quantization field data. For example,the artifact removal stage 606 may change the neighborhood size of thedeblocking filter 600 based on the adaptive quantization field data. Inanother example, the artifact removal stage 606 may use a greater weightfor the values within the original frame when performing the filtering,such as to err on the side of preserving more of the original framedata.

The deblocking filter 600 may be a directional filter such that blockingartifacts removed by the artifact removal stage 606 may be filteredbased on a filtering direction. In particular, the deblocking filter 600(e.g., at the artifact removal stage 606 or another stage (not shown)preceding the artifact removal stage 606) can determine a most invariantdirection within a given area of the frame and filter along that mostinvariant direction. The most invariant direction refers to adirectional line of pixels that has a lowest variation in color, light,or other intensity. The filtering can include calculating average valuesfor the pixels on each side of the directional line and replacingblocking artifacts with those average values.

The artifact removal stage 606 filters along the determined filteringdirection by using the adaptive quantization field data for therespective area or areas of the frame to modulate parameters of thedeblocking filter 600. The directional filter 600 may have a number ofparameters it uses for filtering frame data, including, for example, anon-linearity selection parameter, a filter size parameter, or adirectional sensitivity parameter. The non-linearity selection parameterreflects data or types of data to preserve within the frame (e.g., tonot remove by the filtering). The filter size parameter reflects anumber of pixels that the filter is applied against at a givenoperation. The directional sensitivity parameter reflects a sensitivityof the filtering direction selection.

The adaptive quantization field data is used to modulate some or all ofthe parameters of the deblocking filter 600 based on the quantizationlevels used to encode different areas of the frame. Thus, during theartifact removal stage 606, the adaptive quantization field dataindicates a quantization level used to encode a given area of the frame.The artifact removal stage 606 then uses that quantization levelinformation to control the removal of blocking artifacts within thatgiven area. For example, the artifact removal stage 606 can modulate thenon-linearity selection parameter for filtering a given area of theframe upon determining a fine detail to preserve within the frame (e.g.,a scratch on a painted surface).

Modulating the non-linearity selection parameter includes changing athreshold indicative of a difference between the original frame and thereconstructed frame. The threshold may, for example, reflect a maximumacceptable difference for a given area of the frame based on the valuesof the adaptive quantization field data that are associated with thatgiven area. For example, a greater difference between the original frameand the reconstructed frame within a given area of the frame may beacceptable according to the non-linearity selection parameter where agreater quantization weight is applied in that given area.

The psychovisual modeling stage 608 uses software rules for processingthe frame after filtering at the artifact removal stage 606 based onvisually perceptible qualities of the frame. The software rules of thepsychovisual modeling stage 608 focus on three properties of vision:first, that gamma correction should not be separately applied to everyRGB channel; second, that high frequency changes in blue color data canbe less precisely encoded; and third, that areas including largeramounts of visual noise within a frame can be less precisely encoded.

The first property of vision is driven by the overlap of sensitivityspectra of the cones of the human eye. For example, because there issome relationship between the amount of yellow light seen andsensitivity to blue light, changes in blue color data in the vicinity ofyellow color data can be compressed less precisely. YUV color spaces aredefined as linear transformations of gamma-compressed RGB and aretherefore not powerful enough to model such phenomena. The secondproperty of vision is driven by the color receptors of the retina of thehuman eye. In particular, the human eye has lower spatial resolution inblue than in red and green, and the retina has almost no blue receptorsin the high-resolution area. The third property of vision is definedbased on a relationship between visibility and proximal visual activity.That is, the visibility of fine structures in an area of an image maydepend on the amount of visual activity in the vicinity of that area.

Although the software rules of the psychovisual modeling stage 608 aredescribed with reference to three properties of vision, other numbers ofproperties of vision, other rules related to video or image encoding orperceptibility, or a combination thereof may be used to define orotherwise configure the software rules of the psychovisual modelingstage 608.

The software rules of the psychovisual modeling stage 608 can thus beused to determine a number of blocking artifacts remaining within someor all of the areas of the frame. The psychovisual modeling stage 608next compares that number of blocking artifacts to a psychovisual modelthreshold reflecting a maximum number of visually perceptible artifactsto include in the frame. The value of the psychovisual model thresholdmay be configured by default or set empirically, such as by iteratingthe deblocking filter 600 over N frames. If the psychovisual modelthreshold is exceeded such that the number of blocking artifactsremaining within the frame after the filtering at the artifact removalstage 606 is too large, the psychovisual modeling stage 608 sends thefiltered frame and associated data to the parameter adjustment stage610. Otherwise, the filtered frame 604 is output.

The psychovisual model threshold may also or instead reflect a maximumacceptable difference between the original frame and the frame afterprocessing at the artifact removal stage 606. For example, thepsychovisual modeling stage 608 can use the psychovisual model thresholdas a basis for comparing coefficient or pixel values within a specificarea of the original frame and the frame after processing at theartifact removal stage 606. If that comparison indicates that thecoefficient or pixel value differences between those frames is less thanthe psychovisual model threshold, the differences are preserved.However, if that comparison indicates that those differences are notless than the psychovisual model threshold, the psychovisual modelingstage 608 may cause the frame to undergo further filtering at theartifact removal stage 606.

The parameter adjustment stage 610 adjusts some parameter used by theartifact removal stage 606 before returning the frame to the artifactremoval stage 606 for further filtering. The parameter adjustment stage610 may, for example, adjust one or more values of the adaptivequantization field data. For example, based on the results of thepsychovisual modeling stage 608, the parameter adjustment stage 610 canincrease or decrease the adaptive quantization field data values for agiven area of the frame, such as to correspondingly increase or decreasethe amount of quantization within that given area.

In another example, the parameter adjustment stage 610 may adjust one ormore of the parameters of the deblocking filter 600 itself. For example,based on the visually perceptible qualities of the filtered frame, suchas may be determined as a result of the psychovisual modeling stage 608,the parameter adjustment stage 610 can adjust the non-linearityselection parameter of the deblocking filter 600 for certain areas ofthe frame.

Adjustments to the adaptive quantization field data may be limited by anerror level definition representing a maximum quantization error for theframe. For example, where an adjustment to the adaptive quantizationfield data causes the total quantization error for the frame to exceedthe error level definition, that adjustment is either discarded oroffset by a corresponding adjustment to another area of the frame. Forexample, where a determination is made to adjust a first area of theframe by increasing the quantization weight for that area by X, andwhere that increase causes the total quantization error to exceed theerror level definition, a corresponding determination is also made toadjust a second area of the frame by decreasing the quantization weighttherefore by X.

Adjustments determined or otherwise made at the parameter adjustmentstage 610 are looped back to the artifact removal stage 606 for furtherfiltering. The deblocking filter 600 may iterate multiple times throughsome or all of the artifact removal stage 606, the psychovisual modelingstage 608, or the parameter adjustment stage 610 before the finallyfiltered frame 604 is output. Iterating through the filtering of theframe using the deblocking filter 600 can include producing a model forthe deblocking filter 600. For example, the deblocking filter 600 caninclude functionality for learning the types of filtering applied to andthe types of adjustments that are made for a given frame. When anotherframe is received for filtering, the deblocking filter 600 may use thelearned model to make multiple filtering applications or adjustments ata time, such as to improve processing speeds.

Implementations of the deblocking filter 600 may differ from what isshown and described with respect to FIG. 6. In some implementations, thepsychovisual modeling stage 608 and the parameter adjustment stage 610can be external to the deblocking filter 600. For example, thedeblocking filter 600 may only include functionality for performing theoperations of the artifact removal stage 606. The deblocking filter 600may thus output the filtered frame 604 produced using the artifactremoval stage 606 to the psychovisual modeling stage 608, which may thenprocess the filtered frame 604 to determine whether to re-filter thefiltered frame 604 using the deblocking filter 600.

In some such implementations, the psychovisual modeling stage 608 canprocess the reconstructed frame and adaptive quantization field data 602before the artifact removal stage 606. For example, the psychovisualmodeling stage 608 can process the reconstructed frame and adaptivequantization field data 602 to determine a number of blocking artifactsto remove from the reconstructed frame and the locations of thoseblocking artifacts within the frame. The artifact removal stage 606 thenremoves those blocking artifacts and outputs the filtered frame 604. Theparameter adjustment stage 610 can then receive the filtered frame 604output from the deblocking filter 600 and process the filtered frame 604to determine whether to adjust values of the adaptive quantization fielddata and re-filter the filtered frame 604 using the adjusted adaptivequantization field data.

Where a determination is made to adjust the adaptive quantization fielddata and re-filter the filtered frame 604, the parameter adjustmentstage 610 accordingly adjusts the respective values of the adaptivequantization field data. The parameter adjustment stage 610 then sendsthe filtered frame and adjusted adaptive quantization field data back tothe deblocking filter 600 for re-processing by the artifact removalstage 606. Where a determination is made to not adjust the adaptivequantization field data and re-filter the filtered frame 604, theparameter adjustment stage 610 causes the filtered frame 604 to beoutput, such as to a compressed bitstream, for storage, for furtherprocessing, for rendering at a display, or the like.

In some implementations, the performance of the psychovisual modelingstage 608 may be different depending on the type of data being processedby the deblocking filter 600. For example, where the deblocking filter600 is processing image data (e.g., such that the reconstructed frame isa reconstructed image), the psychovisual modeling stage 608 may use alarger threshold for determining visual perceptibility, as the filteredframe 604 will be affixed when displayed. However, where the deblockingfilter 600 is processing video data (e.g., such that the reconstructedframe is a reconstructed video frame), the psychovisual modeling stage608 may use a smaller threshold for determining the visualperceptibility, as the filtered frame 604 will only be displayed for avery short amount of time.

In some implementations, the deblocking filter 600 may filter the frameother than by determining and using a most invariant direction. Forexample, a machine learning algorithm may be used to determine anoptimal filtering direction to use at the artifact removal stage 606. Inanother example, data indicative of the filtering direction to use maybe explicitly signaled to a decoder from an encoder, such as within aframe header.

FIG. 7 is a block diagram representing portions of a frame 700. Theframe may be a video frame, for example, the video frame 306 shown inFIG. 3A, or an image frame, for example, the image frame 312 shown inFIG. 3B. As shown, the frame 700 includes four 64×64 blocks 710, in tworows and two columns in a matrix or Cartesian plane. In someimplementations, a 64×64 block may be a maximum coding unit, N=64. Each64×64 block may include four 32×32 blocks 720. Each 32×32 block mayinclude four 16×16 blocks 730. Each 16×16 block may include four 8×8blocks 740. Each 8×8 block 740 may include four 4×4 blocks 750. Each 4×4block 750 may include 16 pixels, which may be represented in four rowsand four columns in each respective block in the Cartesian plane ormatrix.

The pixels may include information representing an image captured in theframe 700, such as luminance information, color information, andlocation information. In some implementations, a block, such as a 16×16pixel block as shown, may include a luminance block 760, which mayinclude luminance pixels 762; and two chrominance blocks 770, 780, suchas a U or Cb chrominance block 770, and a V or Cr chrominance block 780.The chrominance blocks 770, 780 may include chrominance pixels 790. Forexample, the luminance block 760 may include 16×16 luminance pixels 762and each chrominance block 770, 780 may include 8×8 chrominance pixels790 as shown. Although one arrangement of blocks is shown, anyarrangement may be used. Although FIG. 7 shows N×N blocks, in someimplementations, N×M blocks may be used. For example, 32×64 blocks,64×32 blocks, 16×32 blocks, 32×16 blocks, or any other size blocks maybe used. In some implementations, N×2N blocks, 2N×N blocks, or acombination thereof, may be used.

In some implementations, coding the frame 700 may include orderedblock-level coding. Ordered block-level coding may include coding blocksof a frame in an order, such as raster-scan order, wherein blocks may beidentified and processed starting with a block in the upper left cornerof the frame, or portion of the frame, and proceeding along rows fromleft to right and from the top row to the bottom row, identifying eachblock in turn for processing. For example, the 64×64 block in the toprow and left column of a frame may be the first block coded and the64×64 block immediately to the right of the first block may be thesecond block coded. The second row from the top may be the second rowcoded, such that the 64×64 block in the left column of the second rowmay be coded after the 64×64 block in the rightmost column of the firstrow.

In some implementations, coding a block of the frame 700 may includeusing quad-tree coding, which may include coding smaller block unitswithin a block in raster-scan order. For example, the 64×64 block shownin the bottom left corner of the portion of the frame 700 may be codedusing quad-tree coding wherein the top left 32×32 block may be coded,then the top right 32×32 block may be coded, then the bottom left 32×32block may be coded, and then the bottom right 32×32 block may be coded.Each 32×32 block may be coded using quad-tree coding wherein the topleft 16×16 block may be coded, then the top right 16×16 block may becoded, then the bottom left 16×16 block may be coded, and then thebottom right 16×16 block may be coded.

Each 16×16 block may be coded using quad-tree coding wherein the topleft 8×8 block may be coded, then the top right 8×8 block may be coded,then the bottom left 8×8 block may be coded, and then the bottom right8×8 block may be coded. Each 8×8 block may be coded using quad-treecoding wherein the top left 4×4 block may be coded, then the top right4×4 block may be coded, then the bottom left 4×4 block may be coded, andthen the bottom right 4×4 block may be coded. In some implementations,8×8 blocks may be omitted for a 16×16 block, and the 16×16 block may becoded using quad-tree coding wherein the top left 4×4 block may becoded, then the other 4×4 blocks in the 16×16 block may be coded inraster-scan order.

In some implementations, coding the frame 700 may include encoding theinformation included in an original, or input, frame by, for example,omitting some of the information in the original frame from acorresponding encoded frame. For example, the coding may includereducing spectral redundancy, reducing spatial redundancy, reducingtemporal redundancy, or a combination thereof.

In some implementations, reducing spectral redundancy may include usinga color model based on a luminance component (Y) and two chrominancecomponents (U and V or Cb and Cr), which may be referred to as the YUVor YCbCr color model, or color space. Using the YUV color model mayinclude using a relatively large amount of information to represent theluminance component of a portion of a frame, and using a relativelysmall amount of information to represent each corresponding chrominancecomponent for the portion of the frame. For example, a portion of aframe may be represented by a high-resolution luminance component, whichmay include a 16×16 block of pixels, and by two lower resolutionchrominance components, each of which represents the portion of theframe as an 8×8 block of pixels. A pixel may indicate a value, forexample, a value in the range from 0 to 255, and may be stored ortransmitted using, for example, eight bits. Although this disclosure isdescribed in reference to the YUV color model, another color model maybe used.

In some implementations, reducing spatial redundancy may includetransforming a block into the frequency domain using, for example, adiscrete cosine transform. For example, a unit of an encoder, such asthe transform stage 404 shown in FIG. 4, may perform a discrete cosinetransform using transform coefficient values based on spatial frequency.

In some implementations, reducing temporal redundancy may include usingsimilarities between frames to encode a frame using a relatively smallamount of data based on one or more reference frames, which may bepreviously encoded, decoded, and reconstructed frames of the videostream. For example, a block or pixel of a current frame may be similarto a spatially corresponding block or pixel of a reference frame. Insome implementations, a block or pixel of a current frame may be similarto block or pixel of a reference frame at a different spatial location,and reducing temporal redundancy may include generating motioninformation indicating the spatial difference, or translation, betweenthe location of the block or pixel in the current frame andcorresponding location of the block or pixel in the reference frame.

In some implementations, reducing temporal redundancy may includeidentifying a portion of a reference frame that corresponds to a currentblock or pixel of a current frame. For example, a reference frame, or aportion of a reference frame, which may be stored in memory, may besearched to identify a portion for generating a predictor to use forencoding a current block or pixel of the current frame with maximalefficiency. For example, the search may identify a portion of thereference frame for which the difference in pixel values between thecurrent block and a prediction block generated based on the portion ofthe reference frame is minimized, and may be referred to as motionsearching. In some implementations, the portion of the reference framesearched may be limited. For example, the portion of the reference framesearched, which may be referred to as the search area, may include alimited number of rows of the reference frame. In an example,identifying the portion of the reference frame for generating apredictor may include calculating a cost function, such as a sum ofabsolute differences (SAD), between the pixels of portions of the searcharea and the pixels of the current block.

In some implementations, the spatial difference between the location ofthe portion of the reference frame for generating a predictor in thereference frame and the current block in the current frame may berepresented as a motion vector. The difference in pixel values betweenthe predictor block and the current block may be referred to asdifferential data, residual data, a prediction error, or as a residualblock. In some implementations, generating motion vectors may bereferred to as motion estimation, and a pixel of a current block may beindicated based on location using Cartesian coordinates as f_(x,y).Similarly, a pixel of the search area of the reference frame may beindicated based on location using Cartesian coordinates as r_(x,y). Amotion vector (MV) for the current block may be determined based on, forexample, a SAD between the pixels of the current frame and thecorresponding pixels of the reference frame.

Although described herein with reference to matrix or Cartesianrepresentation of a frame for clarity, a frame may be stored,transmitted, processed, or any combination thereof, in any datastructure such that pixel values may be efficiently represented for aframe or image. For example, a frame may be stored, transmitted,processed, or any combination thereof, in a two-dimensional datastructure such as a matrix as shown, or in a one-dimensional datastructure, such as a vector array. In an implementation, arepresentation of the frame, such as a two-dimensional representation asshown, may correspond to a physical location in a rendering of the frameas an image. For example, a location in the top left corner of a blockin the top left corner of the frame may correspond with a physicallocation in the top left corner of a rendering of the frame as an image.

As described above, the frame 700 may be a frame of a video sequence, orit may be an image. Regardless of whether the frame 700 represents imagedata or video data, the frame 700 includes a single picture to encode ordecode. The picture included in the frame 700 may have different typesof detail in different areas. An area of a frame as described hereinrefers to an M×N-sized region of the frame, where M and N may be thesame or a different value. For example, an area of a frame may be asingle block (e.g., an 8×8 block) within the frame. In another example,an area of a frame may be multiple blocks within the frame. In yetanother example, different areas of a frame to encode may be differentsized blocks (e.g., some 8×8, some 4×4, some 16×16, etc.).

For example, the frame 700 may include a picture of a forest with a skyover the forest. The forest may be shown in the lower two 64×64 blocks710, while the sky is shown in the upper two 64×64 blocks 710. The dataincluded in the lower two 64×64 blocks 710 may include more detail thanthe data included in the upper two 64×64 blocks 710. For example, thedepiction of the forest may include a number of small details forbranches or leaves of trees along with other plant and/or animal lifepresent in the forest. In contrast, the depiction of the sky may merelyreflect gradient shades of blue. As such, the majority of theinformation in the frame 700 is located in the lower two 64×64 blocks710.

Encoding or decoding the frame 700 can include quantizing the differentareas of the frame 700 (e.g., the different blocks 710, 720, 730, 740,750) by applying different weights to the quantization value for theframe 700 based on the data included in those areas. For example, afirst area of an image that includes more information (e.g., greaterdetail) may be quantized using a smaller weight than a second area thatincludes less information (e.g., lesser detail). The informationcontained within the first area will be quantized less than theinformation contained within the second area. This results in a smallerloss of information within the first area than within the second area.

With the example in which the frame 700 shows a forest and a sky, theadaptive quantization field data for the frame 700 can be produced toreflect that greater quantization is used for the information within theblocks representing the sky and lesser quantization is used for theinformation within the blocks representing the forest. Moreparticularly, values of the adaptive quantization field data for theblocks including the sky information reflect that those blocks arequantized using a larger weight than the blocks including the forestinformation. The values of the adaptive quantization field data for allfour of the 64×64 blocks 710 may be controlled by an error leveldefinition represents the maximum quantization error resulting from theencoding of the frame 700.

Techniques for encoding or decoding video frames are now described withrespect to FIGS. 8 and 9. FIG. 8 is a flowchart diagram of an example ofa technique 800 for encoding or decoding a video frame or an image frameusing a spatially adaptive quantization-aware deblocking filter. FIG. 9is a flowchart diagram of an example of a technique 900 for iterativelyfiltering a video frame or an image frame according to adaptivequantization field data.

One or both of the technique 800 or the technique 900 can beimplemented, for example, as a software program that may be executed bycomputing devices such as the transmitting station 102 or the receivingstation 104 shown in FIG. 1, or otherwise by the computing device 200shown in FIG. 2. For example, the software program can be or otherwiseinclude an encoder, such as the encoder 400 shown in FIG. 4, or adecoder, such as the decoder 500 shown in FIG. 5.

The software program can include machine-readable instructions that maybe stored in a memory such as the memory 204 or the secondary storage214 shown in FIG. 2, and that, when executed by a processor, such as theprocessor 202 shown in FIG. 2, may cause the computing device to performone or both of the technique 800 or the technique 900.

The technique 800 and/or the technique 900, or an encoder and/or decoder(e.g., the encoder 400 and/or the decoder 500) used to perform thetechnique 800 and/or the technique 900, can be implemented usingspecialized hardware or firmware (e.g., an integrated circuit). Asexplained above, some computing devices may have multiple memories orprocessors, and the operations described in the technique 800 and thetechnique 900 can be distributed using multiple processors, memories, orboth.

For simplicity of explanation, the technique 800 and the technique 900are each depicted and described as a series of steps or operations.However, the steps or operations in accordance with this disclosure canoccur in various orders and/or concurrently. Additionally, other stepsor operations not presented and described herein may be used.Furthermore, not all illustrated steps or operations may be required toimplement a technique in accordance with the disclosed subject matter.

Referring first to FIG. 8, the technique 800 for encoding or decoding avideo frame or an image frame using a spatially adaptivequantization-aware deblocking filter is shown. The technique 800 can beperformed for encoding a video frame or an image, such as to abitstream. Alternatively, the technique 800 can be performed fordecoding an encoded video frame or an encoded image, such as from abitstream. For example, during encoding, the technique 800 can beperformed by the reconstruction path (e.g., the stages 410, 412, 414,and 416) of the encoder 400 shown in FIG. 4. In another example, duringdecoding, the technique 800 can be performed by the stages 504, 506,510, and 512 of the decoder 500 shown in FIG. 5.

At 802, quantized transform coefficients associated with a video frameor image (either hereafter referred to as a frame) to encode or decodeare dequantized and inverse transformed. The quantized transformcoefficients represent pixel values of the original video frame ororiginal image after those pixel values are transformed and quantizedduring encoding. The quantized transform coefficients are coefficientsof blocks of the video frame or image. Decoded blocks are produced bydequantizing and inverse transforming the quantized transformcoefficients. The decoded blocks may, for example, be derivativeresidual blocks, such as where the frame is a video frame. At 804, thedecoded video blocks are reconstructed into a reconstructed frame.

At 806, the reconstructed frame is filtered according to adaptivequantization field data associated with the frame. The adaptivequantization field data may be produced at a quantization stage of anencoder or received by a decoder, such as within a bitstream. Filteringthe reconstructed frame according to the adaptive quantization fielddata includes applying a deblocking filter to the reconstructed frame toremove some number of blocking artifacts from the reconstructed frame.The number of blocking artifacts to be removed by the deblocking filteris controlled by the adaptive quantization field data. A first filteredframe is produced as a result of filtering the reconstructed frame.

The performance of the deblocking filter may controlled or otherwiseconfigured using one or more parameters of the deblocking filter. Assuch, applying the deblocking filter to the reconstructed frameaccording to the adaptive quantization field data can include applyingsome or all of the filter parameters to the reconstructed frame, wherethose filter parameters are defined based on the adaptive quantizationfield data. The parameters may, for example, include a non-linearityselection parameter, a filter size parameter, a directional sensitivityparameter, or the like, or a combination thereof. Furthermore, giventhat the filter parameters are defined based on the adaptivequantization field data, the adaptive quantization field data can beused to modulate one or more parameters of the deblocking filter.

Using the adaptive quantization field data to module parameters of thedeblocking filter can include modulating a non-linearity selectionparameter of the deblocking filter according to the adaptivequantization field data to determine to preserve data within thereconstructed frame. For example, the non-linearity selection parametercan be modulated for determining which types of data to preserve withinthe reconstructed frame. As a result, the coefficients used to representsuch types of data within the reconstructed frame are not processed bythe deblocking filter.

For example, the non-linearity selection parameter can be modulatedaccording to a psychovisual model. The psychovisual model can indicatethat certain types of data may be more or less visually perceptible.Thus the non-linearity selection parameter can be modulated based on thepsychovisual model to cause more visually perceptible data within theframe to be preserved and to cause less visually perceptible data withinthe frame to be subject to modification.

The adaptive quantization field data indicates to the deblocking filterthe areas of the reconstructed frame that were processed usingrelatively higher or relatively lower quantization values. As describedabove, the adaptive quantization field data represents weights appliedto quantization values used to encode the blocks of the frame. Thus, thequantization values applied at local areas of the reconstructed frameare used to guide the processing of the deblocking filter.

For example, the reconstructed frame may include a first area that wasprocessed during encoding using a relatively high quantization value anda second area that was processed during encoding using a relatively lowquantization value. The deblocking filter may thus use the adaptivequantization field data for the reconstructed frame to apply arelatively stronger filter to the first area of the reconstructed frameand to apply a relatively weaker filter to the second area. As a result,the deblocking filter removes a greater number of blocking artifactsfrom the first area than from the second area.

At 808, a number of blocking artifacts remaining within the firstfiltered frame is determined. Determining the number of blockingartifacts remaining within the first filtered frame includes processingdata of the first filtered frame using a psychovisual model. Thepsychovisual model may, for example, be a model configured to processvideo frame or image data based on human perceptibility. Implementationsand examples for determining to adjust adaptive quantization field dataare further described below with respect to FIG. 9.

At 810, the adaptive quantization field data is adjusted. Adjusting theadaptive quantization field data includes changing (e.g., increasing ordecreasing) the weight applied to the quantization values for at leastsome of the areas of the first filtered frame. For example, adjustingthe adaptive quantization field data can include increasing a value ofthe adaptive quantization field data, where the value is associated withone or more of the decoded blocks used to produce the reconstructedframe. For example, the value can reflect a weight applied to aquantization value in a particular one of those decoded blocks, thevalue resulting from applying that weight to that quantization value, oranother value reflected by the adaptive quantization field data.

The adjustments to the adaptive quantization field data can be limitedby data associated with the frame being encoded or decoded. For example,increases to the adaptive quantization field data may be limited by anerror level definition associated with the frame. The error leveldefinition represents the maximum quantization error resulting from theencoding of the frame. As such, the adaptive quantization field data maynot be increased in a way that causes the quantization error for theframe to exceed the error level definition. In another example,decreases to the adaptive quantization field data may be limited by afile size definition for the frame (e.g., during encoding operations).As such, the adaptive quantization field data may not be decreased in away that causes the total file size of the resulting encoded frame toexceed the file size definition.

At 812, the first filtered frame produced by the earlier filtering ofthe reconstructed frame is re-filtered according to adjusted adaptivequantization field data. Re-filtering the first filtered frame accordingto the adjusted adaptive quantization field data includes applying thedeblocking filter to the first filtered frame to remove some number ofblocking artifacts from the first filtered frame. The number of blockingartifacts to be removed by the deblocking filter is controlled by theadjusted adaptive quantization field data. A second filtered frame isproduced as a result of filtering the first filtered frame. There-filtering can be performed in the same or substantially the same wayas the earlier filtering of the reconstructed frame.

In some implementations where the technique 800 is performed fordecoding an encoded video frame or an encoded image, the technique 800includes decoding syntax data associated with the encoded video frame orthe encoded image, such as from a bitstream. The syntax data includesthe quantized transform coefficients of the encoded blocks of theencoded video frame or of the encoded image. The syntax data alsoincludes the adaptive quantization field data.

The syntax data may be some or all of the data stored in the bitstreamor another data store and which is associated with the encoded videoframe or the encoded image. For example, the syntax data may include thequantized transform coefficients for some or all of the encoded blocksof the encoded video frame or encoded image. In another example, thesyntax data can include metadata, such as from a frame header for theencoded video frame or from an image header for the encoded image.

In some implementations, the technique 800 includes determining whetherto adjust at the adaptive quantization field data prior to adjusting theadaptive quantization field data. For example, the determination can bemade by comparing the number of blocking artifacts remaining within thefirst filtered frame to a threshold, such as a psychovisual modelthreshold.

In some implementations, the technique 800 includes outputting thesecond filtered frame produced by re-filtering the first filtered frameusing the adjusted adaptive quantization field. For example, when thetechnique 800 is performed for encoding a video frame or an image, thesecond filtered frame can be output to a compressed bitstream, such asthe compressed bitstream 420 shown in FIG. 4, or to storage for latertransmission, such as to the receiving station 104. In another example,when the technique 800 is performed for decoding an encoded video frameor an encoded image, the second filtered frame can be output to anoutput video stream, such as the output video stream 516 shown in FIG.5, or for rendering, such as at the display 218 shown in FIG. 2.

In some implementations, the technique 800 includes further adjustingthe adaptive quantization field data. For example, responsive tore-filtering the reconstructed frame (e.g., to produce second filteredframe data), the adaptive quantization field data may be furtheradjusted, such as based on a number of blocking artifacts within thesecond filtered video frame. For example, a determination can be madethat the first filtered frame data produced by the first filteringincludes too many blocking artifacts (e.g., based on a threshold valueassociated with a psychovisual model). The further adjustment to theadaptive quantization field data can be based on that determination.

Subsequent to the further adjustment of the adaptive quantization fielddata, the deblocking filter can be used to further re-filter thereconstructed frame data, such as to produce a third filtered frame. Thethird filtered frame may then be output, such as to a compressedbitstream, to a compressed image storage, or for further processing(e.g., during encoding), or to an output video stream or for imagerendering (e.g., during decoding). In some implementations, the thirdfiltered frame can be output along with the second filtered frame. Insome implementations, the third filtered frame can be output instead ofthe second filtered frame.

Referring next to FIG. 9, the technique 900 for iteratively filtering avideo frame or an image frame according to adaptive quantization fielddata is shown. The technique 900 can be performed during encodingoperations for encoding a video frame or an image, such as to abitstream. Alternatively, the technique 900 can be performed duringdecoding operations for decoding an encoded video frame or an encodedimage, such as from a bitstream. For example, during encoding, thetechnique 900 can be performed using the deblocking filter 600 shown inFIG. 6 (when included in an encoder) or the deblocking filter of thedeblocking filter stage 416 shown in FIG. 4. In another example, duringdecoding, the technique 900 can be performed using the deblocking filter600 (when included in a decoder) or the deblocking filter of thedeblocking filter 512 shown in FIG. 5.

At 902, frame data and adaptive quantization field data are received.The frame data may, for example, be a reconstructed frame, such as whichmay be produced based on dequantized and inverse transformed video orimage coefficients. The adaptive quantization field data representsweights applied to quantization values used to quantize those video orimage coefficients. At 904, the frame data is filtered according to theadaptive quantization field data. Implementations and examples forfiltering frame data according to adaptive quantization field data aredescribed above with respect to FIG. 8. At 906, a number of blockingartifacts remaining within the filtered frame data is identified.

At 908, a determination is made as to whether the number of blockingartifacts remaining within the filtered frame data exceeds apsychovisual model threshold. The psychovisual model threshold reflectsa maximum number of visually perceptible blocking artifacts that can bepresent within a frame without the frame incurring a loss in visualquality. As such, determining whether the number of blocking artifactsremaining within the filtered frame data exceeds the psychovisual modelthreshold includes using the psychovisual model to determine that thenumber of blocking artifacts remaining within the filtered frame dataexceeds a maximum number of visually perceptible blocking artifactsallowed within the filtered frame.

At 910, responsive to a determination that the number of blockingartifacts remaining within the filtered frame data exceeds thepsychovisual model threshold, one or more values of the adaptivequantization field data are increased, such as to cause morequantization to occur within the one or more corresponding areas of theframe. The technique 900 then returns to 904, where the filtered framedata is further filtered according to the increase value or increasedvalues of the adaptive quantization field data.

At 912, responsive to a determination that the number of blockingartifacts within the filtered frame data does not exceed thepsychovisual model threshold, the filtered frame data are output. Forexample, where the technique 900 is performed to encode the frame, thefiltered frame data is output from a reconstruction path of the encoder(e.g., the stages 410, 412, 414 shown in FIG. 4). In another example,where the technique 900 is performed to decode the frame, the filteredframe data is output to an output video stream (e.g., the output videostream 516 shown in FIG. 5) to a post-filtering stage (e.g., the stage514) before the output video stream.

In some implementations, after identifying a number of blockingartifacts remaining within the filtered frame data, the technique 900includes determining whether a file size for the filtered frame dataincluding that number of remaining blocking artifacts exceeds a filesize threshold. The file size threshold reflects a file size definitionfor the frame data, such as a maximum total file size for an encodedform of the frame data. Determining the file size for the filtered framedata can include encoding the filtered frame data as a temporary encodedframe and identifying the total storage requirements for that temporaryencoded frame.

Responsive to determining that the file size for the filtered frame dataexceeds the file size threshold, one or more values of the adaptivequantization field data are decreased, such as to cause lessquantization to occur within the one or more corresponding areas of theframe. In some implementations, the technique 900 may then return to 904for further filtering of the filtered frame data.

FIG. 10 is an illustration of examples of reproductions of an originalvideo frame or image 1000A using different filtering or filter-lesstechniques. The original video frame or image 1000A represents a videoframe or image before it is encoded and/or decoded, such as using theencoding and decoding system 100 shown in FIG. 1. A first reproduction1000B represents the original video frame or image 1000A after encodingand/or decoding without use of a deblocking filter. A secondreproduction 1000C represents the original video frame or image 1000Aafter encoding and/or decoding with the use of a constant deblockingfilter. A third reproduction 1000D represents the original video frameor image 1000A after encoding and/or decoding with the use of aspatially adaptive quantization-aware deblocking filter, for example,the deblocking filter 600 shown in FIG. 6.

The aspects of encoding and decoding described above illustrate someexamples of encoding and decoding techniques and hardware componentsconfigured to perform all or a portion of those examples of encodingand/or decoding techniques. However, it is to be understood thatencoding and decoding, as those terms are used in the claims, could meancompression, decompression, transformation, or other processing orchanging of data.

The word “example” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“example” is not necessarily to be construed as being preferred oradvantageous over other aspects or designs. Rather, use of the word“example” is intended to present concepts in a concrete fashion. As usedin this application, the term “or” is intended to mean an inclusive “or”rather than an exclusive “or.” That is, unless specified otherwise orclearly indicated otherwise by the context, the statement “X includes Aor B” is intended to mean any of the natural inclusive permutationsthereof. That is, if X includes A; X includes B; or X includes both Aand B, then “X includes A or B” is satisfied under any of the foregoinginstances.

In addition, the articles “a” and “an” as used in this application andthe appended claims should generally be construed to mean “one or more,”unless specified otherwise or clearly indicated by the context to bedirected to a singular form. Moreover, use of the term “animplementation” or the term “one implementation” throughout thisdisclosure is not intended to mean the same implementation unlessdescribed as such.

Implementations of the transmitting station 102 and/or the receivingstation 104 (and the algorithms, methods, instructions, etc., storedthereon and/or executed thereby, including by the encoder 400 and/or thedecoder 500 and including using the technique 800 and/or the technique900) can be realized in hardware, software, or any combination thereof.The hardware can include, for example, computers, intellectual propertycores, application-specific integrated circuits, programmable logicarrays, optical processors, programmable logic controllers, microcode,microcontrollers, servers, microprocessors, digital signal processors,or any other suitable circuit. In the claims, the term “processor”should be understood as encompassing any of the foregoing hardware,either singly or in combination. The terms “signal” and “data” are usedinterchangeably. Further, portions of the transmitting station 102 andthe receiving station 104 do not necessarily have to be implemented inthe same manner.

The transmitting station 102 or the receiving station 104 may beimplemented using a general purpose computer or general purposeprocessor with a computer program that, when executed, carries out anyof the respective methods, algorithms, and/or instructions describedherein. In addition, or alternatively, for example, a special purposecomputer/processor may be utilized, which can include other hardware forcarrying out any of the methods, algorithms, or instructions describedherein.

Some or all of implementations of this disclosure can take the form of acomputer program product accessible from, for example, a computer-usableor computer-readable medium. A computer-usable or computer-readablemedium can be any device that can, for example, tangibly contain, store,communicate, or transport the program for use by or in connection withany processor. The medium can be, for example, an electronic, magnetic,optical, electromagnetic, or semiconductor device. Other suitablemediums are also available.

The above-described embodiments, implementations, and aspects have beendescribed in order to facilitate easy understanding of this disclosureand do not limit this disclosure. On the contrary, this disclosure isintended to cover various modifications and equivalent arrangementsincluded within the scope of the appended claims, which scope is to beaccorded the broadest interpretation as is permitted under the law so asto encompass all such modifications and equivalent arrangements.

1. A method for decoding an encoded frame from a bitstream, the methodcomprising: decoding syntax data associated with the encoded frame fromthe bitstream, the syntax data including quantized transformcoefficients of encoded blocks of the encoded frame and adaptivequantization field data representing weights applied to quantizationvalues used to encode the encoded blocks; dequantizing and inversetransforming the quantized transform coefficients of the encoded blocksto produce decoded blocks; reconstructing the decoded blocks into areconstructed frame; applying a deblocking filter to the reconstructedframe according to the adaptive quantization field data to produce afirst filtered frame; using a psychovisual model to determine that anumber of blocking artifacts within the first filtered frame exceeds amaximum number of visually perceptible blocking artifacts allowed withinthe first filtered frame; responsive to determining that the number ofblocking artifacts exceeds the number of visually perceptible blockingartifacts, adjusting the at least some of the adaptive quantizationfield data based on the number of blocking artifacts to produce adjustedadaptive quantization field data; applying the deblocking filter to thefirst filtered frame according to the adjusted adaptive quantizationfield data to produce a second filtered frame; and outputting the secondfiltered frame for display.
 2. The method of claim 1, wherein applyingthe deblocking filter to the reconstructed frame according to theadaptive quantization field data to produce the first filtered framecomprises: using the adaptive quantization field data to modulate one ormore parameters of the deblocking filter.
 3. The method of claim 2,wherein using the adaptive quantization field to modulate the one ormore parameters of the deblocking filter comprises: modulating anon-linearity selection parameter of the one or more parametersaccording to the adaptive quantization field data to determine topreserve data within the reconstructed frame.
 4. The method of claim 2,wherein the one or more parameters include one or more of anon-linearity selection parameter, a filter size parameter, or adirectional sensitivity parameter. 5-6. (canceled)
 7. The method ofclaim 1, wherein adjusting the at least some of the adaptivequantization field data based on the number of blocking artifactscomprises: increasing a value of the adaptive quantization field data,the value associated with one or more of the decoded blocks, theincreasing limited by an error level definition associated with theencoded frame.
 8. The method of claim 1, further comprising: subsequentto producing the second filtered frame, adjusting at least one value ofthe adjusted adaptive quantization field data based on a number ofblocking artifacts within the second filtered frame to produce furtheradjusted adaptive quantization field data; and applying the deblockingfilter to the second filtered frame according to the further adjustedadaptive quantization field to produce a third filtered frame, whereinoutputting the second filtered frame for display comprises: outputtingthe third filtered frame for display.
 9. An apparatus for decoding anencoded frame from a bitstream, the apparatus comprising: a processorconfigured to execute instructions stored in a non-transitory storagemedium to: decode encoded blocks of the encoded frame to produce decodedblocks; reconstruct the decoded blocks into a reconstructed frame;filter the reconstructed frame according to adaptive quantization fielddata associated with the encoded frame to produce a first filteredframe; subsequent to filtering the reconstructed frame, use apsychovisual model to adjust at least some of the adaptive quantizationfield data based on a determination that a number of blocking artifactswithin the first filtered frame exceeds a maximum number of visuallyperceptible blocking artifacts allowed within the first filtered frame;subsequent to adjusting the at least some of the adaptive quantizationfield data, filter the first filtered frame according to the adaptivequantization field data to produce a second filtered frame; and outputthe second filtered frame for display.
 10. The apparatus of claim 9,wherein the instructions include instructions to: decode the adaptivequantization field data from the bitstream prior to decoding the encodedblocks, the adaptive quantization field data representing weightsapplied to quantization values used to encode the encoded blocks. 11.The apparatus of claim 9, wherein the instructions to filter thereconstructed frame according to the adaptive quantization field dataassociated with the encoded frame to produce the first filtered frameinclude instructions to: use the adaptive quantization field data tomodulate one or more parameters of a deblocking filter used to filterthe reconstructed frame.
 12. The apparatus of claim 11, wherein theinstructions to use the adaptive quantization field data to modulate theone or more parameters of the deblocking filter used to filter thereconstructed frame include instructions to: modulate a non-linearityselection parameter of the one or more parameters according to theadaptive quantization field data to determine to preserve data withinthe reconstructed frame.
 13. The apparatus of claim 11, wherein the oneor more parameters include one or more of a non-linearity selectionparameter, a filter size parameter, or a directional sensitivityparameter.
 14. (canceled)
 15. The apparatus of claim 11, wherein theinstructions to use the psychovisual model to adjust the at least someof the adaptive quantization field data include instructions to:increase a value of the adaptive quantization field data, the valueassociated with one or more of the decoded blocks, the increase limitedby an error level definition associated with the encoded frame.
 16. Amethod, comprising: applying first filter parameters to a reconstructedframe to produce a first filtered frame, wherein the first filterparameters are defined based on adaptive quantization field dataassociated with the reconstructed frame; using a psychovisual model todetermine that a number of blocking artifacts within the first filteredframe exceeds a maximum number of visually perceptible blockingartifacts allowed within the first filtered frame; responsive todetermining that the number of blocking artifacts exceeds the number ofvisually perceptible blocking artifacts, adjusting the first filterparameters to produce second filter parameters, wherein the adjustingincludes increasing at least one value of the adaptive quantizationfield data; applying the second filter parameters to the first filteredframe to produce a second filtered frame; and outputting the secondfiltered frame for display.
 17. The method of claim 16, wherein thereconstructed frame is produced based on an encoded frame, wherein theadaptive quantization field data represents weights applied toquantization values used to encode encoded blocks of the encoded frame.18. The method of claim 17, wherein the increasing of the at least onevalue of the adaptive quantization field data is limited by an errorlevel definition associated with the encoded frame.
 19. (canceled) 20.The method of claim 16, wherein the first filter parameters areparameters of a deblocking filter, wherein applying the first filterparameters to the reconstructed frame to produce the first filteredframe comprises: modulating a non-linearity selection parameter of theparameters according to the adaptive quantization field data todetermine to preserve data within the reconstructed frame.
 21. Themethod of claim 1, wherein the maximum number of visually perceptibleblocking artifacts allowed within the first filtered frame is based on apsychovisual model threshold, wherein a value of the psychovisual modelthreshold is empirically configured.
 22. The method of claim 21, whereinthe psychovisual model threshold corresponds to a maximum acceptabledifference between the reconstructed frame and the first filtered frame.23. The apparatus of claim 9, wherein the maximum number of visuallyperceptible blocking artifacts allowed within the first filtered frameis based on a psychovisual model threshold, wherein a value of thepsychovisual model threshold is empirically configured.
 24. The methodof claim 16, wherein the maximum number of visually perceptible blockingartifacts allowed within the first filtered frame is based on apsychovisual model threshold, wherein a value of the psychovisual modelthreshold is empirically configured.